1. Field of the Invention
The invention relates to apparatus and methods for determining and correcting geometric distortion in electronic imaging systems and, particularly, to a novel method for using electronic imaging in taking distortion-corrected images.
2. Description of the Related Art
Electronic imaging systems, such as digital cameras and infrared (IR) focal plane array imaging systems, have many photogrammetric applications in geological survey, traffic accident investigations, diagnosis and treatment in medicine, robotics, automatic manufacturing, military surveillance, and weapons targeting systems.
With reference to FIG. 1, there depicted a diagram of a digital camera. Light from the surface of object 1 passing through an optical lens 2, forms image on a two dimensional (2-D) intensity sensor array 4 in the 2-D image detector array unit 6 when the shutter 3 is open. The sensing array 4 comprises multiple opto-electronic sensors. The intensity signals excited by impinging lights then are recorded in an on-chip memory buffer 5, which is also in the 2-D image detector array 6, and then are output via a data link 7 to another memory 8 for post processing.
Ideally, image forming in a camera can be modeled as "pin-hole" projections. FIG. 2 depicts an object point A at object plane 1 to be projected to image point a on the image plane 3 according to the pin-hole camera model. With the presence of distortion caused by lens and the opto-electronic sensor array, which hereafter will be simply termed as geometric distortion, the actual location of the image of the object point will be at a' on the same image plane. In general, the electronic imaging system components, mainly the lens and the sensor array, defines a mapping on the image plane:
L:(x,y).fwdarw.(x',y')
where (x, y) is the coordinate of the ideal location a on the image plane in the camera intrinsic coordinate system according to the pin-hole projection, and (x', y'), is the actual coordinate of actual image point a' of A found on the image in the same intrinsic coordinate system. The coordinate of the actual image point a' is a function of the ideal projection a of A: x'=l.sub.x (x, y) and y'=l.sub.y (x, y). The 2-D mapping L(x, y)=(l.sub.x (x, y), l.sub.y (x, y)) from ideal locations of all visible scene points in the field of view to their actual image registration locations will be termed the geometric distortion mapping, or simply distortion mapping, of the electronic imaging system.
The cameras for photogrammetric measurements are required to have nearly perfect geometric optic performances. Even slight geometric distortion might cause great error in measurements.
Traditionally, high quality lenses or combination lenses are used to reduce the distortion. Today, distortions in high quality combination lenses have been reduced to such negligible amounts that they are often referred to as distortion-free lenses.
In a film camera, the sensing is achieved through chemical changes of continuously spread coating material in direct response to the projected light. In this case, the correction of distortion caused by the lens must be a process happening before the light reaches the film. Once the light reaches the film, the geometric distortion will appear in the photograph and it will be too late to correct it. In this case, the optical means including using high quality lens or combination lenses appears the only viable method to correct the geometric distortion in the images. The optical means of lens perfection will be referred as the pre-imaging correction of geometric distortion. The distortion-free lenses are difficult to fabricate and very expensive.
In photogrammetry, there are available means of calibration and correction to overcome the effect of geometric distortion. For example, in the textbook by Wolf, Paul R. (1974) Elements of Photogrammetry, McGRAW-HILL BOOK COMPANY, New York, and earlier in the paper by Duane C. Brown ("Advanced Methods for Calibration of Metric Cameras", for 1969 Symposium on Computational Photogrammetry), methods for calibrating lens parameters and making measurement corrections were described. These correction methods do not change the photographs. The correction process only change the assignments of the coordinates of the image points on the photographs. The image points with corrected coordinates can be used in photogrammetric calculations with reduced distortion error. The computer process of calibration and correction will be referred as the post-imaging correction of lens distortion.
Based on the models of particular types of lens imperfection with regard to geometric optics and the calibrated parameters describing such imperfection, numerical scheme can be derived to calculate the amount of distortion error at each location of a photograph. No matter how precise the calibration is, and how accurate the lens distortion model is, the post-imaging correction process does not improve image geometric quality. It only brings benefits to photogrammetric measurements.
Besides the lens distortion, in electronic imaging systems there exists another important source of geometric distortion, the sensor array irregularities. The above mentioned methods for reducing the geometric distortions in image measurements, by improving the lenses or by accurate calibration and analytical corrections, are only effective for lens distortions.
Unlike the conventional optical camera wherein the photographic film is the image recording media, in a digital camera, with reference to FIG. 1, a two-dimensional (2-D) array 4 of light sensing elements is used to detect the spatial light intensity distribution on the image plane. Many types of 2-D image sensing array have developed over the last 30 years. For example, charge coupled device (CCD) array is the widely used one in current digital cameras. The sensor arrays of the electronic imaging systems usually are not as flat and smoothly surfaced as the films. Once the sensor array is made, it will be difficult to change its geometric shape. Thus, besides the lens distortion, the electronic imaging systems usually have another source of geometric distortion: the sensor array distortion. The analytical correction schemes based on lens optical paths, such as D. C. Brown's, are not adequate for electronic imaging systems.
In recent years, there emerge some sampling-based methods of geometric distortion correction for using digital cameras in measurement in laboratories. The sampling-based method performs a set of experiments on image areas where image distortion is to be corrected. The set of experiments is performed to collect a sample of the geometric distortion mapping.
More specifically, the sampling-based method uses some distinguishable feature points with known object location which allows calculation of ideal, undistorted position on the image. The "ideal positions" and the actual registrations of the feature point on the image provide samples of the geometric distortion mapping. With the samples of the distortion mapping collected over a region of image via experiments, appropriate interpolation scheme then can be applied to make approximation of the geometric distortion mapping. The measurement corrections then can be made according to the approximate local geometric distortion mapping.
The sampling-based method depends upon neither lens distortion models nor calibration methods, therefore it can be applied to the geometric distortion correction in general ways. The disadvantage of this method is that the experiments involves great effort and usually are limited in accuracy as well as in the number and spacing of the sample points, if expensive and accurate equipments are not available to the imaging system user.
A digital camera, or in general an electronic imaging system, has two major parts. The optical part is quite similar to that of an conventional camera. The sensing part is different. It is an opto-electronic sensor system. In recent years, novel technique closely related to the opto-electronic sensor technologies called computational sensors or smart sensors has emerged. Smart sensor methods are useful in enhancing the performance of sensors. A smart sensor consists a transducer combined with signal conditioning and other circuitry to produce output information signals that have been logically operated upon to increase the value of the information. At a higher level, the conversion-level smart sensors detect specific hard-to-measure parameters and covert that measurement to electrical signal. Sensors with environmental compensation and communication have further enhanced capabilities.
Smart sensing techniques have been applied to imaging sensor systems. For examples, focal plane array of circuits have been applied to imaging sensors which performs image processing while sensing. Some more advanced vision chips called "artificial retina" further implement machine vision algorithms on the sensing chip. These chips not only having functions of optical sensing, but also having functions of noise reduction, DC (direct current) component suppression, and contrast enhancing. In recent years, DSP (digital signal processor) has become a powerful tool in the smart sensor technologies.
In another related area, computer associative memory, which associate one data item to another data item for a large assembly of data set, are developed for various purposes, ranging from database applications to pattern recognition. The forms of associative memory can be computer memory look-up tables or artificial neural nets.
The current art of smart sensor techniques for electronic imaging is basically limited to enhancing the quality of electrical signals. The application of associative memory to imagery is mainly for feature extraction and pattern recognition, which are particular visual information processing tasks. These processing methods does not help for geometric distortion correction: Neither the local signal enhancement nor the local image feature extraction or pattern recognition processes can provide direct help for reducing image geometric distortion and thus enhancing the geometric quality of images.